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(57) A color scannerless range imaging system in- 
cludes an illumination system for illuminating the scene 
with modulated illumination of a predetermined modu- 
lation frequency and an image capture section posi- 
tioned In an optical path of the reflected illumination from 
the scene for capturing a plurality of Images Including 
(a) at least one range image corresponding to the re- 
flected modulated illumination and including a phase de- 
lay corresponding to the distance of objects in the scene 
from the range imaging system, and (b) at least one oth- 
er image of reflected unmodulated illumination corre- 
sponding to color in the scene. The image capture sec- 
tion includes a color filter array arranged with a first type 
of color filter that preferentially transmits the reflected 
modulated illumination and one or more other color f 11- 
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ters that preferentially transmit the reflected unmodulat- 
ed illumination, an image intensifier receiving the reflect- 
ed illumination and including a modulating stage for 
modulating the reflected modulated illumination from 
the scene, thereby generating the range information, 
and an image responsive element for capturing an out- 
put of the image intensifier, including the range image 
and the other image of reflected unmodulated image 
light corresponding to color in the scene. The image In- 
tensifier is structured with channels that generally cor- 
respond to the color filter array such that the intensifier 
provides the range image from channels corresponding 
to the first color filter and the other image corresponding 
to color in the scene from channels corresponding to the 
other color filters. 
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Description 

FIELD OF THE il^VENTION 

5 [0001] The present invention relates to the field of three-dimensional image capture and in particular to the capture 
of a color texture image in conjunction with a scannerless range imaging system. 

BACKGROUND OF THE INVENTiOIS) 

10 [0002] Standard image capture systems will capture images, such as photographic images, that are two-dimensional 
representations of the three-dimensional world. In such systems, projective geometry best models the process of trans- 
fomriing the three-dimensional real world into the two-dimensional Images. In particular, much of the information that 
is lost in the transformation is in the distance between the camera and image points in the real world. Methods and 
processes have been proposed to retrieve or record this infomiation. Some methods, such as one based on a scanner 

IS from Cyberware, Inc., use a laser to scan across a scene. Variations in the reflected light are used to estimate the 
range to the object. However, these methods require the subject to be close (e.g., within 2 meters) to the camera and 
are typically slow. Stereo imaging Is a common example of another process, which is fast on capture but requires 
solving the "correspondence problem", that is, the problem of finding corresponding points in the two images. This can 
be difficult and limit the number of pixels having range data, due to a limited number of feature points that are suitable 

^o for the correspondence processing. 

[0003] Another method described in U.S. Patent 4,935,61 6 (and further described in the Sandia Lab News, vol. 46, 
No. 19, September 16, 1994) provides a scannerless range imaging system using either an amplitude-modulated high- 
power laser diode or an array of amplitude-modulated light emitting diodes (LEDs) to completely illuminate a target 
scene. Conventional optics confine the target beam and image the target onto a receiver, which includes an integrating 

25 detector array sensor having hundreds of elements In each dimension. The range to a target is determined by measuring 
the phase shift of the reflected light from the target relative to the amplitude-modulated carrier phase of the transmitted 
light. To make this measurement, the gain of an image intensifier (in particular, a micro-channel plate) within the receiver 
is modulated at the same frequency as the transmitter, so the amount of light reaching the sensor (a charge-coupled 
device) is a function of the range-dependent phase difference. A second image is then talcen without receiver or trans- 

30 mitter modulation and Is used to eliminate non-range-carrying intensity Information. Both captured images are regis- 
tered spatially, and a digital processor is used to operate on these two frames to extract range. Consequently, the 
range associated with each pixel is essentially measured simultaneously across the whole scene. 
[0004] The preferred method of estimating the range in the '61 6 patent uses a pair of captured Images, one Image 
with a destructive interference caused by modulating the Image intensifier, and the other with the image intensifier set 

35 at a constant voltage. However, a more stable estimation method uses a series of at least three Images, each with 
modulation applied to the image intensifier, as described in commonly assigned copending application Serial No. 
09/342,370, entitled "Method and Apparatus for Scannertess Range Image Capture Using Photographic Film" and filed 
June 29, 1999 in the names of Lawrence Allen Ray and Timothy P. Mathers. In that application, the distinguishing 
feature of each image is that the phase of the image intensifier modulation is unique relative to modulation of the 

40 illuminator. If a series of n Images are to be collected, then the preferred arrangement Is for successive images to have 
a phase shift of — radians (where n is the number of images) from the phase of the previous image. However, this 
specific shift is n<ft required, albeit the phase shifts need to be unique. The resultant set of images is referred to as an 
Image bundle. The range at a pixel location is estimated by selecting the intensity of the pixel at that location in each 
image of the bundle and performing a best fit of a sine wave of one period through the points. The phase of the resulting 

45 best-fitted sine wave is then used to estimate the range to the object based upon the wave-length of the illumination 
frequency. 

[0005] An image intensifier operates by converting photonic energy into a stream of electrons, amplifying the energy 
of the electrons and then converting the electrons back Into photonic energy via a phosphor plate. One consequence 
of this process is that color information is lost. Since color is a useful property of images for many applications, a means 

so of acquiring the color Information that is registered along with the range Information is extremely desirable. One ap- 
proach to acquiring color is to place a dichromatic mirror in the optical path before the microchannel plate. Following 
the mirror a separate image capture plane (i.e., a separate image sensor) is provided for the range portion of the 
camera and another image capture plane (another sensor) is provided for the color texture capture portion of the 
camera. This is the approach taken by 3DV Technology with their 2-Cam product. Besides the added expense of two 

55 image capture devices, there are additional drawbacks in the need to register the two image planes precisely, together 
with alignment of the optical paths. Another difficulty is collating image pairs gathered by different sources. 
[0006] Another approach is described in detail in commonly assigned copending application Serial No. 09/572,522. 
entitled "Method and Apparatus for a Color Scannerless Range Image System" and filed May 1 7. 2000 in the names 
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of Lawrence Allen Ray and Louis R. Gabello. In this system, a primary optical path is established for directing image 
light toward a single image responsive element. A beamsplitter located in the primary optical path separates the image 
light into two channels, a first channel including an infrared component and a second channel including a color texture 
component. One of the channels continues to traverse the primary optical path and the other channel traverses a 

5 secondary optical path distinct from the primary path. A modulating element is operative in the first channel to receive 
the infrared component and a modulating signal, and to generate a processed infrared component with phase data 
indicative of range infomriation. An optical network is provided In the secondary optical path for recombining the sec- 
ondary optical path into the primary optical path such that the processed infrared component and the color texture 
component are directed toward the single image responsive element. While this approach avoids the added expense 

10 of two image capture devices, there continues to be the need to register the two image planes precisely, together with 
alignment of the optical paths. 

[0007] Another approach is to capture an image bundle by using two interchangeable optical assemblies: one optical 
assembly for the phase image portion and a separate optical element for the color texture image portion. This approach 
is described in detail in commonly assigned copending application Serial No. 09/451 ,823, entitled "Method and Appa- 
15 ratus for a Color Scannerless Range Image System" and filed November 30, 1 999 in the names of Lawrence Allen 
Ray, Louis R. Gabello and Kenneth J. Repich. The drawback of this approach is the need to switch lenses and the 
possible misregistration that might occur due to the physical exchange of lens elements. There is an additional draw- 
back in the time required to swap the two optical assemblies, and the effect that may have on the spatial coincidence 
of the images. 

20 [0008] A scannerless range imaging camera may operate either as a digital camera or a camera utilizing film. In the 
case of a film based system there are some other requirements, particularly registration requirements, that need to be 
met. These requirements and means for satisfying them are described in the aforementioned copending application 
Serial No. 09/342,370. As mentioned above, the drawback of such a camera system, including a film-based system, 
is its inability to capture a color image. 

25 [0009] In most conventional digital imaging systems, the ability to determine color is accomplished by means of a 
color filter array (CFA) arranged in front of the image responsive device, such as a charge-coupled device (CCD). The 
CFA overcomes the CCD's inherent inability to discriminate color, i.e., the CCD is basfcally a monochrome device. The 
manufacturing and use of a CFA is well known, see, e.g., U.S. Patent 4,315,978, entitled "Solid-state color imaging 
device having a color filter array using a photocrosslinkable barrier". This filter system, which is different from the 

30 standard filter in that the filter is an array of small color filters, is employed by a large number of digital cameras, for 
example the DCS-series of cameras manufactured by Eastman Kodak Company. Typically, the array is comprised of 
an array of 2 x 2 pixel sub-filters. The sub-filter blocks have two diagonal cells capable of filtering green light, and one 
other cell is capable of filtering red light and the third is capable of filtering blue light. Upon completing the image capture 
a full color pixel is formed by interpolation based upon pixels capturing the desired color. Color filter arrays with more 

35 than three primary colors are also known in the art (see, e.g., U.S. Patent 5,719,074, entitled "Method of making a 
planar color filter array for CCDs from dyed and mordant layers"). In particular, this method allows for filters with any 
repeatable pattern of color sensitive dyes. Each of these patents are incorporated herein by reference. 
[0010] It is known, in certain cases, to apply color filters to an image intensifier. In U.S. Patent No. 4,374,325, entitled 
"Image intensifier arrangement with an in situ fonned output filter", an image intensifier device is provided with color 

40 filters on its input and output surfaces so as to intensify a color image without losing the color content. Each filter 
consists of an array of red, green and blue elements and these elements are precisely aligned in both input and output 
filters to avoid degradation of the color content. A method of producing the output filter in situ is described to provide 
the required accuracy of alignment. In U.S. Patent No. 5,233,1 83, entitled "Color image intensifier device and method 
for producing same", a four color system is specified In which a color image intensifier device includes infra-red filters 

15 in an RGB input matrix and a narrow band output filter is assigned to represent I R infomnation In the RGB output matrix. 
In each of these cases, the output image from the intensifier is adapted for human viewing; thus the output image 
needs to reconverted back to a color image, and hence the need for a second color filter behind the phospher element 
at the output of the intensifier. In U.S. Patent No. 5,1 61 ,008, entitled "Optoelectronic image sensor for color cameras", 
an image sensor includes an image intensifier arranged between an interiine type semiconductor sensor, coupled to 

50 the output of the intensifier, and a color stripe filter disposed in front of the photocathode such that one color stripe of 
the color stripe filter is associated with one column of light-sensitive elements of the semiconductor sensor. Each of 
these patents are incorporated herein by reference. 

[0011] As mentioned above, the drawback of a range Imaging camera system, including a film-based system, is the 
inability of current designs to capture a color image. What is needed is a convenient camera system that would avoid 
55 the aforementioned limitations and capture ranging information without sacrificing color information that would other- 
wise be available for capture. 
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SUMMARY OF THE INVENTION 

[0012] An object of the Invention is to obtain a color image along with range infomnation for each point on the image. 
[0013] A further object is to obtain such a color image by introducing a color filter array prior to the photo-cathode 

5 on the microchannel plate In the intenslfier, where the color filter array is matched to the spatial pattern of the micro- 
channel plate. Individual cells of the filter array are also arranged to provide a simple means of Interpolation. 
[0014] The present invention Is directed to overcoming one or more of the problems set forth above. Briefly summa- 
rized, according to one aspect of the invention, a color scannerless range imaging system includes an illumination 
system for illuminating the scene with modulated illumination of a predetermined modulation frequency and an Image 

10 capture section positioned in an optical path of the reflected Illumination from the scene for capturing a plurality of 
images thereof, including (a) at least one range image corresponding to the reflected modulated illumination and in- 
cluding a phase delay corresponding to the distance of objects in the scene from the range imaging system, and (b) 
at least one other image of reflected unmodulated illumination corresponding to color in the scene. The image capture 
section includes a color filter array arranged with a first type of color filter that preferentially transmits the reflected 

15 modulated Illumination and one or more other color filters that preferentially transmit the reflected unmodulated illumi- 
nation, an Image Intenslfier receiving the reflected illumination and including a modulating stage for modulating the 
reflected modulated illumination from the scene with the predetemrilned modulation frequency, thereby generating the 
range information, and an image responsive element for capturing an output of the image intenslfier, Including the 
range image corresponding to the reflected modulated Image light and the other image of reflected unmodulated image 

20 light corresponding to color in the scene. The image intenslfier Is structured with channels that generally correspond 
to the color filter array such that the intenslfier provides the range image from channels corresponding to the first color 
filter and the other Image corresponding to color in the scene from channels corresponding to the one or more other 
color filters. 

[0015] The advantage of this invention is that a single image capture system is required, thereby reducing cost, 
25 correlation and Image capture variations. Moreover, the combined range and texture image is color Instead of mono- 
chrome. The system does not require beam-splitters or difficult optical waveguides, and the overall system may be an 
attachment to a standard camera system 



30 



40 



BRIEF DESCRIPTION OF THE DRAWINGS 
[0016] 



FIG. 1 shows the main components of a color scannerless range imaging system In accordance with the invention. 
FIGS. 2A and 2B are diagrams of the color filter array used in the system shown In Figure 1 . 
35 FIG. 3 Is a diagram Illustrating an Image bundle and related data captured by the system shown In Figure 1 . 

FIG. 4 shows the various stages of a film-based range imaging process in accordance with the invention. 
FIG. 5 shows the main components of a film-based color scannerless range imaging system in accordance with 
the Invention. 

FIG. 6 Is a diagram of hexagonal scanning employed with the film-based range imaging process shown in Figure 4. 



DETAILED DESCRIPTION OF THE INVENTION 



[0017] Because range imaging devices employing laser illuminators and capture devices Including Image intensiflers 
and electronic sensors are well known, the present description will be directed in particular to elements fomriing part 

45 of, or cooperating more directly with, apparatus in accordance with the present Invention. Elements not specifically 
shown or described herein may be selected from those known in the art. Certain aspects of the embodiments to be 
described may be provided in software. Given the system as shown and described according to the invention in the 
following materials, software not specifically shown, described or suggested herein that is useful for implementation 
of the Invention Is conventional and within the ordinary skill In such arts. 

so [0018] Referring first to Figure 1 , acolor range Imaging system 10 is shown as a laser radar that is used to illuminate 
a scene 12 and then to capture an image bundle comprising a minimum of three images of the scene 1 2. An illuminator 
1 4 emits a beam of electromagnetic radiation whose frequency is controlled by a modulator 1 6. Typically the illuminator 
14 is a laser device which includes an optical diffuser in order to effect a wide-field illumination. As will be described 
in greater detail, it is preferable that the modulating laser is an IR source. This is also for eye-safety issues and to 

55 operate in the spectral region of maximal response by the capture system. The modulator 16 provides an amplitude 
varying sinusoidal modulation. The modulated illumination source is modeled by: 
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Ut) = A s\n{2nXt) (Eq.1) 

where p.^ is the mean illumination, t| is the modulus of the illumination source, and A. is the modulation frequency applied 
to the illuminator 14. The modulation frequency is sufficiently high (e.g.. 10 MHz) to attain sufficiently accurate range 
estimates. The output beam 18 is directed toward the scene 12 and a reflected beam 20 is directed back toward a 
receiving section 22. As is well known, the reflected beam 20 is a delayed version of the transmitted output beam 18, 
with the amount of phase delay being a function of the distance of the scene 12 from the range imaging system. 
(Importantly for this invention, the reflected beam 20 will always include unmodulated illumination, for instance, ambient 
or studio color illumination reflected from objects in the scene. The illuminator 14 may also include a separate light 
source (not shown) for emitting unmodulated illumination; then the reflected unmodulated illumination from the scene 
may include at least some of the emitted unmodulated illumination.) 

[0019] The receiving section 22 includes an image intensifier 26 having a microchannel plate 30. a photocathode 24 
on a front, input face of the microchannel plate 30 and a phosphor screen on a rear, output face of the microchannel 
plate 30. A color filter array 28 is positioned in front of the photocathode 24 in order to provide the intensifier 26 with 
the capability of producing color images. As is well known, colors of the visible spectrum may be reproduced by com- 
binations of different sets of colors, such as red, green and blue or cyan, magenta and yellow. Moreover, it is well 
known that the sensitivity of an Image intensifier is partly derived from the fact that the photocathode Is mostly respon- 
sive to near-infrared radiation (400 - 900 nanometers), part of which is invisible to the human eye. Accordingly, since 
the advantageous effect of the invention is obtained by confining the range data to the infra-red radiation, the modulation 
frequency of the illumination is restricted to the infra-red region, and the visible region separated by the color fitter array 
is therefore substantially unaffected by the modulation. Therefore, referring to Figures 2A and 2B, the color filter array 
28 is shown to comprise a pattern of four distinct color filters, e.g., red, blue, green and infrared filters. As shown In 
Figure 2A, the filter 28 is arranged Into a hexagonal lattice designed to match the channel pattern of the microchannel 
plate 28. The red, green and blue channels are designed to limit the amount of light from neighboring bands from 
passing through the filter. The infrared filter element is intended to pass infrared light, but may also pass red light as 
well. As shown in Figure 2B, the arrangement of the color filter elements within the lattice provide an additional desirable 
property. Given a color filter element 200, the six neighboring elements do not share the same color filtering property, 
and there are precisely two neighboring color filter elements 222, 224, 226 for each color. For instance, if the given 
color element 200 is a green color element, then color filter elements 222 could be blue color elements, color filter 
elements 224 could be red color elements and color filter elements 226 could be infra-red color elements. Each two 
elements of the same color are thus on directly opposite sides of the central color filter element, which provides a 
desirable property when it comes to interpolation of the missing colors for each color element, that is, each missing 
color for a central color filter element may be interpolated from two adjacent color filter elements of the same color. 
35 [0020] The reflected beam 20 passes through the color filter an^ay 28 and strikes the photocathode 24 within the 
image intensifier 26; the Infra-red portion of the beam 20 produces a modulated electron stream proportional to the 
input amplitude variations (hereinafter referred as the IR component), while the red, green and blue portions of the 
beam 20 produce an electron stream proportional to the red, green and blue content of the scene (hereinafter referred 
as the color component). Since the photocathode 24 does not separately process the colors in the scene, the electron 
stream created at this point is essentially monochromatic, I.e., except for its spatial relationship to the color filter array, 
the color wavelength infomiation in the electron stream Is lost at this time. The IR component output of the image 
intensifier 26 Is modeled by: 

45 M(t) = + y sin(27cX0 (Eq.2) 

where \if^ Is the mean intensification, y is the modulus of the intensification and X is the modulation frequency applied 
to the intensifier 26. The purpose of the image intensifier is not only to intensify both components of the image, but 
also to act as a modulating shutter for the IR component. Accordingly, the image intensifier 26 is connected to the 
modulator 16, such that the electron stream strikes the intensifier 26 and the IR component Is modulated by an IR 
modulating signal from the modulator 16. The electron stream, including both components, is then amplified through 
secondary emission by the microchannel plate 30. The Intensified electron stream bombards a phosphor screen 32, 
which converts the energy into a visible light image. While the color wavelength infomiation has been lost, the spatial 
correspondence of the visible light image to the color filter array 28 has been maintained. Some of this visible light 
image represents (spatially) the color components, and another portion represents the IR component. The intensified 
light image signal is captured by a capture mechanism 34, such as a charge-coupled device (CCD). The electronic 
output of the CCD is then separated by conventional processing techniques into the IR component and the several 



40 



so 



55 



OCID: <EP 1178333A2J_> 



5 



10 



15 



20 



25 



30 



40 



45 



50 



55 



EP 1 178 333 A2 

color components (i.e., RGB components). The IR component of the captured image signal is applied to a range 
processor 36 to determine the phase delay at each point in the scene. The phase delay term co of an object at a range 
p meters is given by: 

€i> = ?^mod27i (Eq.3) 

where c is the velocity of light in a vacuum. Consequently, the reflected light at this point Is modeled by: 

R(t) = ^^ = K sin(2jtXf + 00) (Eq.4) 

where k is the modulus of illumination reflected from the object. The pixel response P at this point is an integration of 
the reflected light and the effect of the Intensification: 

R{t)M{t)dt = l^tf^M + cos(6>) (Eq. 5) 



[0021] The improved approach described in the aforementioned Serial Number 09/342,370, which Is incorporated 
herein by reference, collects at least three phase images (referred to as an image bundle). As shown in relation to 
Figure 3, the notion of an image bundle 90 Is central to the range estimation method. The image bundle 90 includes 
a combination of images captured by the system as well as information pertinent to the individual images and infor- 
mation common to all the images. The image bundle contains two types of images: range images 92 related to the IR 
component portion of the process and a color image 94, commonly referred to as the texture image. Common infor- 
mation 96 in the image bundle 90 would typically include the number of images In the bundle (three or more) and the 
modulation frequency utilized by the camera system. Other information might be the number of horizontal and vertical 
pixels in the Images, and/or data related to camera status at the time of the image capture. Image specific information 
will include the phase offset 1 ...N used for each (1 ...N) of the individual range images 92. The image bundle 90 includes 
a minimum of three such images, each of which are monochrome. Each of the range images 92 records the effect of 
a distinct phase offset applied to either the illuminator 14 or to the image intensifier 26. The additional color image 94 
is an Image developed from color channels obtained from the red, green and blue color filters, and does not contain 
range capture components. Although this is a color image, it is preferably, but not necessarily, the same size as the 
range images 92. 

[0022] In the present approach, the phase of the intensifier 26 is shifted relative to the phase of the illuminator 14, 
and each of the phase images has a distinct phase offset. Forthis purpose, the range processor36 is suitably connected 
to control the phase offset of the modulator 1 6, as well as the average illumination level and such other capture functions 
as may be necessary. If the image intensifier 26 (or laser illuminator 1 4) is phase shifted by 9/, the pixel response from 
equation (5) becomes: 



= 2 ji. ^ft ^71 + K7CY cos(<o + e /) (Eq . 6) 

[0023] It is desired to extract the phase term u) from the expression. However, this term is not directly accessible 
from a single IR component image. In equation (6) there are three unknown values and the fomn of the equation is 
quite simple. As a result, mathematically only three samples (from three IR component images) are required to retrieve 
an estimate of the phase term, which is equivalent to the distance of an object in the scene from the imaging system 
10. Therefore, a set of three IR component images captured with unique phase shifts is sufficient to detemnine w. For 
simplicity, the phase shifts are given by 6^ = 2nk/3\ k= 0,1 ,2. In the following description, an image bundle shall be 
understood to constitute a collection of IR component images which are of the same scene, but with each image having 
a distinct phase offset obtained from the modulation applied to the intensifier 26. It should also be understood that an 
analogous analysis can be performed by phase shifting the illuminator 14 instead of the intensifier 26. If an image 
bundle comprising more than three images is captured, then the estimates of range can be enhanced by a least squares 
analysis using a singular value decomposition (see, e.g., W.H. Press, B.P. Flannery, S.A. Teukolsky and W.T. Vetteriing, 
Numerical Recipes (the Art of Scientific Computing) . Cambridge University Press, Cambridge, 1986). 
[0024] If images are captured with n>3 distinct phase offsets of the Intensifier (or laser or a combination of both) 
these images form an image bundle. Applying Equation (6) to each IR component image in the image bundle and 
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expanding the cosine teim 

(I.e., Pf = 2\ii}ii^ + iciry(cos(o>)cos(e/)- sln(a))sln(G,))) results In the following systenn of linear equations in n unknowns 
at (sach point: 
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^1 cos^, -sin^i ^ 
1 cos ^2 -sin ^2 

1 cos^, -sin^„ 



A 



(Eq. 7) 



75 



where A =2n/>i^7t, A2 = lory cos o), and A3 = KTrysinu). This system of equations is solved by a singular value decom- 
position to yield the vector A = [A^, A2, A3, p. Since this calculation is carried out at every (x,y) location in the image 
bundle, A is really a vector image containing a three element vector at every point. The phase term 10 is computed at 
each point using a four-quadrant arctangent calculations: 



20 



CO = tan' (A3.A2) 



(Eq.8) 



The resulting collection of phase values at each point forms the phase image. Once phase has been determined, range 
r can be calculated by: 
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r=o> 



(Eq.9) 



Equations (1)-(9) thus describe a method of estimating range using an image bundle with at least three images'(i.e., 
n=3) corresponding to distinct phase offsets of the intensifier and/or laser 

[0025] Various techniques may be used to manufacture and attach the color filter to an intensifier tube. For instance, 
in the aforementioned U.S. Patent No. 5,233,183, which is incorporated herein by reference, the filters are screen 
printed onto a thin glass substrate, e.g., a wafer, which becomes a glass layer that Is sandwiched between a faceplate 
and the photocathode. In U.S. Patent No. 4,374,325, which is incorporated herein by reference, an array of colored 
hexagons or squares are deposited in situ across the surface of a fiber optic plate attached to the photocathode in 
order to provide the required accuracy of alignment. U.S. Patent No. 5,742, 1 15, entitled "Color Image Intensifier Device 
and Method for Producing Same", which is incorporated herein by reference, describes the hexagonal placement 
(Figure 2) and alignment of color filters to an intensifier, where the alignment of input (and output) color filters is im- 
plemented in the course of the regular procedure of imager intensifier manufacturing. In particular, a faceplate incor- 
porating the color filters is adjusted in an operating tube under visual control until in the correct position all chromatic 
moire vanish and the correct monochromatic flat field response will appear in the field of vision. Then the various 
elements are secured in place. Each of these patents also teach the arrangement of a second color filter on the output 
of the intensifier tube. While the present invention has been described in relation to the location of the color filter on 
the front of the intensifier, it is to be understood that the color filter could alternatively be placed on the rear of the 
intensifier. While two color filters are not necessary, it should also be understood that intensifier tubes with color filters 
on both ends would operate successfully with the present invention. The claims are intended to cover these various 
arrangements, regardless of their details of construction and manufacture. 

[0026] As has been described, an intensifier tube comprises a photocathode that converts light into electrons, that 
are then transferred to a array of channels within the microchannel plate. While passing through the channels, the 
electrons are amplified. The channels in the array are typically in the arrangement of a honeycomb, e.g., a hexagonal 
lattice, and are not in spatial alignment with the photosensitive pixel pattern of most CCD imagers. As would be indicated 
by the teaching of the aforementioned U.S. Patent No. 5,161 ,008, an intensifier incorporating a stripe filter may be 
aligned with a conventional rectilinear two-dimensional semiconductor sensor, and therefore this misalignment would 
not necessarily be a critical matter; nonetheless, it may prove desirable in some situations to provide an imager with 
a matching lattice of pixels, which could be accomplished with conventional techniques of the kind invoh/ed in the 
manufacture of CCD, or other types of, imagers . Alternatively, this problem may be avoided if the image that is emitted 
from the microchannel plate can be collected onto monochromatic photographic film. The capture of range data onto 
photographic film has been disclosed by Ray and Mathers in the aforementioned commonly assigned copending ap- 
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plication Serial No. 09/342,370. One fundamental problem of utilizing film is the need to register the images for use in 
the range estimation process. Ray and Mathers teach a method to accomplish this task. 

[0027] Figure 4 shows a film-based embodiment of the present invention. Including an Image capture system 100 
that has the ability to emit modulated illumination and to synchronize a receiver with the same frequency through a 

5 locked phase offset. This system records the images onto photographic film with a geometric pattern of pixel elements, 
as described above. The system 100 captures a plurality of images of a scene 112, with a minimum of three of such 
images. The plurality of images 113, referred to as an image bundle 115, are identical, with the exception that the phase 
offset provided for the IR component for each image in the image bundle is unique. In addition, the system 100 puts 
fiducial markings on the film in order to assure image registration later in the range estimation process. The images In 

10 the image bundle are recorded on a photographic film 114. While any film will work, it is preferred that a film such as 
EASTMAN PLUS-X Reversal Film 7276 from Eastman Kodak Company is used. Such a film will provide the overall 
speed needed for the image capture system, as well as providing excellent material uniformity. The film 114 is then 
developed in a film development unit 1 1 6. Care needs to be taken in order to minimize process non-unlformlties in the 
film development process for successive film frames. The developed film images 113 are then scanned in a scanner 

15 118 and converted into a digital image bundle 119, with each image 1 1 7 being scanned using identical scanning char- 
acteristics. The digital image bundle 119 is then transferred to an image processing computer 120. The digital image 
bundle 119 Is first registered by aligning the fiducial marks on each image 117. This is accomplished by a standard 
pattem matching algorithm. The preferred manner is to select one of the digital images 117 in the digital image bundle 

119 and register all other Images to that Image. Once the images are all registered, information regarding the phase 
20 offsets of each image and the modulation frequency of the camera 100 illumination system are given to the computer 

120 and a registered image bundle 121 results. In addition, a range estimate for each registered pixel is performed 
and stored as part of the Image bundle 121 . 

[0028] Referring now to Figure 5, the camera system 100 Is described in further detail, wherein elements common 
with Figure 1 are shown with the same reference numeral. The camera system 100 is comprised of a number of 

25 subassemblies in addition to the normal assemblies common to most cameras. In particular the camera has an image 
bundle control system 122 that manages the camera, including standard camera functions, the average illumination 
level provided to the film 114, the phase offset of the Image Intensif ler 26, a film advance 1 40, and a camera data-back 
138. The canriera data-back 138, such as a Nikon MF-28, is used to write a pattern between the image areas, which 
Is used later to register the Images, as described In the aforementioned Serial Number 09/342,370. As described in 

30 connection with Figure 1 , the illuminator 14 provides an Illumination source that can be modulated at sufficiently high 
frequency, e.g., lOmHz to attain sufficiently accurate range estimates. Typically, the illuminator 14 is a laser device 
that has an optbal diffuser in order to affect a wide-field illumination. The Illumination does not have to be uniform, but 
does need to have sufficient intensity to generate a distinguishable reflection. The modulation controller 16 controls 
the modulation frequency of the illumination source 14 and ot the image intensifier 26. Part of the function of the 

35 modulation controller 16 Is to introduce the appropriate phase delay to the image Intensifier 26. The light from the 
illumination source 14 is reflected from the scene and returned through a standard camera lens (not shown). The light 
then passes through the color filter array 28. The image Intensifier amplifies the returning light and acts as a modulating 
shutter for the IR component. After passing through the image Intensifier 26, the photons bombard a phosphor screen 
32 which converts the energy into visible light that is directed by a lens 134 upon the film 114. 

40 [0029] As shown in Figures 1 and 5, the image capture device can be a high quality monochrome film 114 or a 
charged-couple device (CCD) 34, preferably with a hexagonal lattice of pixel cells. In the preferred embodiment, the 
color filter array matches the hexagonal pattern of the microchannel plate; consequently, a hexagonal pattern corre- 
sponding spatially to the color filter array is projected upon the image capture device. Thus, the preferred embodiment 
of the system makes use of this hexagonal lattice of pixels at the image plane. Either the CCD would share this ar- 

45 rangement, or in the case of film, a film scanner would be capable of conducting a hexagonal scan of the film. Most 
Image scanners use a rectilinear lattice to scan an Image. However, the regions of the film images that are to be 
converted into pixel values are structured using a hexagonal lattice. Assuming the scanner has been spatially calibrated, 
the preferred method of scanning is to perfomi two separate scans. The reason for this is that the hexagonal lattice 
can be readily decomposed into two rectilinear lattices. Each rectilinear lattice is scanned and the results are then 

50 interieaved in the obvious manner to recover the hexagonal lattice. Figure 6 shows a form of hexagonal scanning 
based on two rectilinear scans, one scan represented by the scanning linelSO and the other scan by a scanning line 
160, that may be used to obtain the range and texture image from the film. The "x" points In the figure describe the 
hexagonal sampling pattems employed by the two scans. Further details regarding hexagonal scanning can be found 
in "The Processing of Hexagonally Sampled Two-Dimensional Signals", by P.M. Mersereau, Proceedings of the IEEE, 

55 vol. 67, no. 8, June 1979, pp. 930 - 949. There is also the potential for variation in where the image is captured onto 
film, and where the scanner anticipates the Image position on the film there is a need to calibrate the scanner position 
based upon the fiducial marks written on the film. This method was described by Ray and Mathers in the aforementioned 
Serial Number 09/342,370. 
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[0030] The technique of interpolating the pixels of the resulting image can be understood in relation to Figure 2B. 
The collected image has by itself no color infomriation. since the preferred embodiment utilizes a CCD or black and 
white film and more importantly because the electrons emitted from themicrochannel plate bombard a phosphor screen 
that provides no color differentiation. However, because of the pattem of the color filter array as projected on the CCD 

5 or film, color can be inferred at individual points on the image plane. The scanned cells from the film or the CCD 
elements are mapped to the color of the filter array element projected upon them. This is the same basic premise as 
used by current digital cameras. Knowledge of the current cell's color filter element and the arrangement of the CFA 
hexagonal pattem shown in Figure 2B determines the pattern of color filter elements of neighboring cells. For Instance, 
a red pixel has two neighboring green pixels, two neighboring blue pixels and two neighboring infrared pixels. The 

10 position of these colors relative to the central pixel is uniform. The pixels of the same color are also arranged to be 
diagonally opposite. As a result, one method of estimating the missing color values is a simple average of the two 
opposing pixels of the same color. Accordingly, the result is that each captured image includes four "color" bands or 
channels, e.g., red, green, blue and IR channels. 

[0031 ] The invention has been described in detail with particular reference to certain preferred embodiments thereof, 
15 but it will be understood that variations and modifications can be effected within the spirit and scope of the invention. 

Claims 

20 1 . A color scannerless range imaging system for capturing both color and range infomnation of a scene, said system 
comprising; 

an illumination system for controllably Illuminating the scene with modulated illumination of a predetermined 
modulation frequency, whereby some of the modulated illumination is reflected from objects in the scene; 

25 an image capture section positioned in an optical path of the reflected illumination from the scene for capturing 

a plurality of images thereof, including (a) at least one range image corresponding to the reflected modulated 
illumination, whereby the modulation of the reflected modulated illumination incorporates a phase delay cor- 
responding to the distance of objects in the scene from the range imaging system, and (b) at least one other 
^ image of reflected unmodulated illumination corresponding to color in the scene, said image capture section 

30 comprising: 

a color fitter array positioned in the optical path of the reflected Illumination and comprised of a predeter- 
mined filter pattem including a first color filter that preferentially transmits the reflected modulated illumi- 
nation and one or more other color filters that preferentially transmit the reflected unmodulated illumination; 

35 an image intensifier receiving the reflected illumination and including a modulating stage for modulating 

the reflected modulated illumination from the scene with the predetemnined modulation frequency, thereby 
generating the range information, wherein the image intensifier has a channel structure that generally 
corresponds to the predetermined pattem of the color filter array such that the intensifier provides the 
range image from channels corresponding to said first color filter and the other image con-esponding to 

^0 color In the scene from channels corresponding to said one or more other color fitters; and 

an image responsive element for capturing an output of the image intensifier, including the range image 
corresponding to the reflected modulated image light and the other image of reflected unmodulated image 
light corresponding to color in the scene. 

2. The range imaging system as claimed in claim 1 wherein the intensifier is a micro-channel plate. 

3. The range imaging system as claimed in claim 1 wherein the image intensifier is interposed in the optical path 
between the image responsive element and the color filter array. 

50 4. The range imaging system as claimed in claim 1 wherein the image responsive element is a photosensitive film. 

5. The range imaging system as claimed In claim 1 wherein the image responsive element is an electronic image 
sensor. 

55 6. The range imaging system as claimed in claim 1 wherein the image responsive element captures a plurality of 
range images corresponding to the reflected modulated illumination, wherein each range image incorporates the 
effect of the predetermined modulation frequency together with a phase offset unique for each image. 
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7. The range Imaging systenn as claimed in claim 1 wherein the illumination system includes a laser illuminator for 
producing the modulated illumination. 

8. The range imaging system as claimed in claim 1 wherein the predetermined modulating frequency is an infra-red 
5 frequency and said first color fitter Is an infra-red color filter. 

9. The range imaging system as claimed in claim 1 wherein said one or more other color filters comprise a set of 
color filters including at least any two filters selected from the group of red, green and blue filters. 

10 10. The range imaging system as claimed in claim 2 wherein the microchannel plate includes channels forming a 
hexagonal lattice structure and the color filter array has a hexagonal pattern that substantially matches the lattice 
structure of the microchannel plate. 
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